376 research outputs found
NP/CMP equivalence: a phenomenon hidden among sparsity models l_{0} minimization and l_{p} minimization for information processing
In this paper, we have proved that in every underdetermined linear system Ax = b, there corresponds a constant p*(A, b) > 0 such that every solution to the l p-norm minimization problem also solves the l0-norm minimization problem whenever 0 <; p <; p*(A, b). This phenomenon is named NP/CMP equivalence
PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matrices
Pipeline parallelism enables efficient training of Large Language Models
(LLMs) on large-scale distributed accelerator clusters. Yet, pipeline bubbles
during startup and tear-down reduce the utilization of accelerators. Although
efficient pipeline schemes with micro-batching and bidirectional pipelines have
been proposed to maximize utilization, a significant number of bubbles cannot
be filled using synchronous forward and backward passes. To address this
problem, we suggest that extra work be assigned to the bubbles to gain
auxiliary benefits in LLM training. As an example in this direction, we propose
PipeFisher, which assigns the work of K-FAC, a second-order optimization method
based on the Fisher information matrix, to the bubbles to accelerate
convergence. In Phase 1 pretraining of BERT-Base and -Large models, PipeFisher
reduces the (simulated) training time to 50-75% compared to training with a
first-order optimizer by greatly improving the accelerator utilization and
benefiting from the improved convergence by K-FAC
The sparsity of underdetermined linear system via lp minimization for 0 < p < 1
The sparsity problems have attracted a great deal of attention in recent years, which aim to find the sparsest solution of a representation or an equation. In the paper, we mainly study the sparsity of underdetermined linear system via lp minimization for 0 0 such that the following conclusions hold when p < γ(A, b): (1) the problem (pp) generates sparser solution as the value of p decreases; (2) the sparsest optimal solution to the problem (pp) is unique under the sense of absolute value permutation; (3) let X1 and X2 be the sparsest optimal solution to the problems (pp1) and (pp2) , respectively, and let X1 not be the absolute value permutation of X2. Then there exist t1,t2 ε [p1,p2] such that X1 is the sparsest optimal solution to the problem (pt) (∀t ε [p1, t1]) and X2 is the sparsest optimal solution to the problem (pt) (∀t ε (t2, p2])
Pulse-duration dependence of high-order harmonic generation with coherent superposition state
We make a systematic study of high-order harmonic generation (HHG) in a
He-like model ion when the initial states are prepared as a coherent
superposition of the ground state and an excited state. It is found that,
according to the degree of the ionization of the excited state, the laser
intensity can be divided into three regimes in which HHG spectra exhibit
different characteristics. The pulse-duration dependence of the HHG spectra in
these regimes is studied. We also demonstrate evident advantages of using
coherent superposition state to obtain high conversion efficiency. The
conversion efficiency can be increased further if ultrashort laser pulses are
employed
- …